Information processing apparatus, system, and method for displaying bio-information or kinetic information

ABSTRACT

An information processing apparatus includes a bio-information obtaining unit configured to obtain bio-information of a subject; a kinetic-information obtaining unit configured to obtain kinetic information of the subject; and a control unit configured to determine an expression or movement of an avatar on the basis of the bio-information obtained by the bio-information obtaining unit and the kinetic information obtained by the kinetic-information obtaining unit and to perform a control operation so that the avatar with the determined expression or movement is displayed.

CROSS REFERENCES TO RELATED APPLICATIONS

This application is a continuation of and claims the benefit under 35U.S.C. § 120 of U.S. patent application Ser. No. 16/291,792, titled“INFORMATION PROCESSING APPARATUS, SYSTEM, AND METHOD FOR DISPLAYINGBIO-INFORMATION OR KINETIC INFORMATION,” filed Mar. 4, 2019, which is acontinuation of and claims the benefit under 35 U.S.C. § 120 of U.S.patent application Ser. No. 15/946,954, titled “INFORMATION PROCESSINGAPPARATUS, SYSTEM, AND METHOD FOR DISPLAYING BIO-INFORMATION OR KINETICINFORMATION,” filed Apr. 6, 2018, now U.S. Pat. No. 10,262,449, which isa continuation of and claims the benefit under 35 U.S.C. § 120 of U.S.patent application Ser. No. 15/389,979, titled “INFORMATION PROCESSINGAPPARATUS, SYSTEM, AND METHOD FOR DISPLAYING BIO-INFORMATION OR KINETICINFORMATION,” filed on Dec. 23, 2016, now U.S. Pat. No. 9,972,116, whichis a continuation of and claims the benefit under 35 U.S.C. § 120 ofU.S. patent application Ser. No. 14/314,385, titled “INFORMATIONPROCESSING APPARATUS, SYSTEM, AND METHOD FOR DISPLAYING BIO-INFORMATIONOR KINETIC INFORMATION,” filed on Jun. 25, 2014, now U.S. Pat. No.9,568,998, which is a continuation of and claims the benefit under 35U.S.C. § 120 of U.S. patent application Ser. No. 12/221,526, titled“INFORMATION PROCESSING APPARATUS, SYSTEM, AND METHOD THEREOF,” filed onAug. 4, 2008, now U.S. Pat. No. 8,797,331, which claims priority to andcontains subject matter related to Japanese Patent Application JP2007-204114, filed in the Japanese Patent

Office on Aug. 6, 2007, all of which are hereby incorporated herein byreference in their entireties.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to information processing apparatuses,systems, and methods thereof, and more particularly, to the technique ofdisplaying avatars in accordance with bio-information and kineticinformation of a user and other persons.

2. Description of the Related Art

Japanese Unexamined Patent Application Publication No. 2006-126891discloses the technique relating to transmitting bio-information over anetwork.

Japanese Unexamined Patent Application Publication Nos. 2006-34803,2006-87829, 2003-79591, and 2004-194996 each describe an apparatus fordisplaying various types of bio-information.

Japanese Unexamined Patent Application Publication No. 2007-11391discloses a map generating system of extracting a user's subjectiveinformation and mapping the information on a map.

SUMMARY OF THE INVENTION

No systems of easily and clearly presenting bio-information and kineticinformation of a user and other persons or situations estimated fromthese items of information have been proposed yet.

In particular, detected results of bio-information such as the heartrate or blood pressure have generally been shown using numerals orgraphs. However, this is targeted at professionally trained people withknowledge in the medical field and is not a simple way of presentingsuch information.

It is desirable to display bio-information and kinetic information of auser and other persons or situations estimated from these items ofinformation in a manner that enables general people other thanprofessionals to easily understand these items of information orsituations estimated therefrom, thereby creating various applications.

An information processing apparatus according to an embodiment of thepresent invention includes the following elements: bio-informationobtaining means for obtaining bio-information of a subject (e.g., aperson); kinetic-information obtaining means for obtaining kineticinformation of the subject; and control means for determining anexpression or movement of an avatar on the basis of the bio-informationobtained by the bio-information obtaining means and the kineticinformation obtained by the kinetic-information obtaining means andperforming a control operation so that the avatar with the determinedexpression or movement is displayed.

The bio-information obtaining means may be a bio-information detectingunit configured to detect bio-information, and the kinetic-informationobtaining means may be a kinetic-information detecting unit configuredto detect kinetic information. That is, the information processingapparatus itself has the function of detecting bio-information andkinetic information.

The bio-information obtaining means and the kinetic-informationobtaining means may be communication units configured to receivebio-information and kinetic information, respectively, from an externalapparatus. That is, the bio-information obtaining means and thekinetic-information obtaining means may obtain bio-information andkinetic information, respectively, using a communication function.

The information processing apparatus may further include display meansfor displaying an image. The control means may perform a controloperation so that the avatar is displayed on the display means.

The information processing apparatus may further includeposition-information obtaining means for obtaining position information;and map-image obtaining means for obtaining map image data. The controlmeans may control, on the basis of the position information obtained bythe position-information obtaining means, the map-image obtaining meansto obtain map image data to display a map image and may perform acontrol operation so that the avatar is displayed at a position inaccordance with the position information on the map image.

The control means may perform, when the bio-information obtaining meansand the kinetic-information obtaining means obtain bio-information andkinetic information, respectively, a display control operation todisplay the avatar with the expression or movement determined on thebasis of the bio-information and the kinetic information. That is, theavatar is displayed in real-time on the basis of the just obtainedbio-information and kinetic information.

The information processing apparatus may further include saving meansfor saving information. The control means may cause, when thebio-information obtaining means and the kinetic-information obtainingmeans obtain bio-information and kinetic information, respectively, thesaving means to save the obtained bio-information and kineticinformation or avatar-related information indicating the expression ormovement of the avatar, the expression or movement being determined onthe basis of the obtained bio-information and kinetic information.

In that case, the control means may read the bio-information and thekinetic information or the avatar-related information from the savingmeans under a predetermined condition and perform a display controloperation so that the avatar is displayed using the read bio-informationand kinetic information or the read avatar-related information. That is,the avatar can be displayed on the basis of bio-information and kineticinformation obtained in the past.

The bio-information may include information indicating at least one ofthe pulse rate, heart rate, electrocardiographic signal, electromyogram,breathing (e.g., the rate and depth of breathing and the amount ofventilation), perspiration, galvanic skin response (GSR), bloodpressure, saturation of pulse oximetry oxygen (SpO2), skin surfacetemperature, brain wave (e.g., α wave, β wave, θ wave, and δ waveinformation), blood flow change (change in the flow of blood such ascerebral blood or peripheral blood detected using near-infraredspectroscopy), temperature, and status of eyes (status of pupils,movement of eyes, blink, etc.).

The kinetic information may be information indicating at least one of astill state, a walking state, a running state, an exercising state(e.g., shaking or jumping), and movement of a body element (e.g., thehead, arm, leg, hand, or finger).

According to another embodiment of the present invention, there isprovided an information processing system including a transmittingapparatus; and an information processing apparatus. The transmittingapparatus includes bio-information detecting means for detectingbio-information of a subject (e.g., a person who is wearing thetransmitting apparatus), kinetic-information detecting means fordetecting kinetic information of the subject, and transmitting means fortransmitting and outputting the bio-information detected by thebio-information detecting means and the kinetic information detected bythe kinetic-information detecting means. The information processingapparatus includes communication means for communicating information,and control means for causing the communication means to receivebio-information and kinetic information, determining an expression ormovement of an avatar on the basis of the bio-information and kineticinformation received by the communication means, and performing acontrol operation so that the avatar with the determined expression ormovement is displayed.

The information processing system may further include a server apparatusincluding storage means for storing bio-information and kineticinformation. The transmitting means of the transmitting apparatus maytransmit the bio-information and the kinetic information to the serverapparatus. The server apparatus may store the transmittedbio-information and kinetic information in the storage means. Thecommunication means of the information processing apparatus may receivethe bio-information and kinetic information stored in the storage meansof the server apparatus.

Alternatively, the communication means of the information processingapparatus may receive avatar-related information indicating theexpression or movement of the avatar, the expression or movement beingdetermined on the basis of the bio-information and kinetic informationstored in the storage means of the server apparatus.

According to another embodiment of the present invention, there isprovided an information processing method including the steps ofobtaining bio-information of a subject (e.g., a person); obtainingkinetic information of the subject; determining an expression ormovement of an avatar on the basis of the obtained bio-information andkinetic information; and performing a control operation so that theavatar with the determined expression or movement is displayed.

According to the embodiments of the present invention, an avatar with anexpression or movement determined on the basis of bio-information andkinetic information is displayed for a user. The avatar is assumed tobe, for example, an image of a personified animal or object or ananimated image of a person. By determining the expression or movement ofthe avatar on the basis of bio-information and kinetic information, theperson's state such as the state of movement, health, or emotion can berepresented.

According to the embodiments of the present invention, the state of aperson (such as a user or another person) serving as a subject can berepresented using the expression or movement of a displayed avatar.Accordingly, a person having no technical background can easily know theperson's or another person's state. By displaying an image representingthe state of a person, various applications can be realized, includingan application for managing the person's health, a communicationsapplication, and an application for enhancing the feeling of delight atviewing images displayed on various apparatuses.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an information display apparatus accordingto an embodiment of the present invention;

FIG. 2 is block diagram of another example of the information displayapparatus according to the embodiment;

FIGS. 3A to 3F are illustrations of avatars according to the embodiment;

FIG. 4 is a flowchart of a process of displaying an avatar according tothe embodiment;

FIGS. 5A and 5B are illustrations of examples of displayed avatarsaccording to the embodiment;

FIGS. 6A and 6B are illustrations of recording states of bio-informationand the like according to the embodiment;

FIG. 7 is a flowchart of a process of reproducing and displaying anavatar according to the embodiment;

FIGS. 8A and 8B are illustrations of examples of reproduced anddisplayed avatars according to the embodiment;

FIG. 9 is a diagram of an information processing system according to theembodiment;

FIG. 10 is a block diagram of a transmitting apparatus according to theembodiment;

FIG. 11 is a block diagram of a server apparatus according to theembodiment;

FIG. 12 is a block diagram of a display apparatus according to theembodiment;

FIG. 13 is a block diagram of another example of the display apparatusaccording to the embodiment;

FIG. 14 is a flowchart of a process of transmitting bio-information andthe like to the server apparatus according to the embodiment;

FIG. 15 is an illustration of storage states of bio-information and thelike in the server apparatus according to the embodiment;

FIG. 16 is a flowchart of a process of transmitting data to the displayapparatus and displaying an image represented by the data according tothe embodiment;

FIG. 17 is a flowchart of another example of the process of transmittingdata to the display apparatus and displaying an image represented by thedata according to the embodiment; and

FIG. 18 is an illustration of an example of displayed avatars of a userand another person.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Embodiments of the present invention are described below. Descriptionsare given in the following order:

1. Displaying Avatar Based on User's Bio- and Kinetic Information 1-1Structure of Information Display Apparatus 1-2 Process of DisplayingAvatar in Real-Time 1-3 Process of Reproducing and Displaying Avatars asState History 2. Displaying Avatar Based on Another Person's Bio- andKinetic Information 2-1 System Configuration 2-2 Structure ofTransmitting Apparatus 2-3 Structure of Server Apparatus 2-4 Structureof Display Apparatus 2-5 System Operation for Displaying Avatar 3.Displaying Avatars Based on User's and Another Person's Bio- and KineticInformation 4. Advantages and Modifications of Embodiment 1. DisplayingAvatar Based on User's Bio- and Kinetic Information 1-1 Structure ofInformation Display Apparatus

Referring to FIG. 1, the structure of an information display apparatus 1is described. The information display apparatus 1 is an apparatuscarried by a user. The information display apparatus 1 displays, on itsdisplay screen, an avatar based on bio- and kinetic information of theuser carrying the information display apparatus 1.

The information display apparatus 1 may be, for example, an apparatuswith a glasses-type display and worn on the head of the user.Alternatively, the information display apparatus 1 may be a watch-typeapparatus and worn on the wrist of the user. Alternatively, theinformation display apparatus 1 may be an apparatus having a shape andsize suitable for being carried by the user, such as a cellular phone ora personal digital assistant (PDA). Further, the information displayapparatus 1 may be provided as an internal function which is included ina cellular phone or a PDA and serves as the information displayapparatus 1.

The information display apparatus 1 includes a system controller 10, abio-sensor unit 11, a kinetic sensor unit 12, a position detecting unit13, a display unit 14, a time-and-date counting unit 15, an operationunit 16, a storage unit 17, and a communication unit 18.

The system controller 10 is implemented by, for example, a microcomputerincluding a central processing unit (CPU), a read-only memory (ROM), arandom access memory (RAM), a non-volatile memory, and an interface. Thesystem controller 10 controls the overall information display apparatus1.

On the basis of an internal operation program, the system controller 10controls the components of the information display apparatus 1 toperform a display operation based on bio- and kinetic information.

The bio-sensor unit 11 detects a user's bio-information. Bio-informationincludes, for example, the pulse rate, heart rate, electrocardiographicsignal, electromyogram, breathing (e.g., the rate and depth of breathingand the amount of ventilation), perspiration, galvanic skin response(GSR), blood pressure, saturation of pulse oximetry oxygen (SpO2), skinsurface temperature, brain wave (e.g., 60 wave, β wave, θ wave, and δwave information), blood flow change (change in the flow of blood suchas cerebral blood or peripheral blood detected using near-infraredspectroscopy), temperature, and status of eyes (status of pupils,movement of eyes, blink, etc.).

For example, the GSR, temperature, skin surface temperature,electrocardiogram response, electromyogram heart rate, pulse rate, bloodflow, blood pressure, brain wave, or perspiration can be detected using,for example, a sensor in contact with the skin of a subject. The brainwave can be detected using a sensor in contact with and worn on the headof a subject.

A sensor that detects the eyes of a user can be implemented by, forexample, an imaging unit that captures an image of the user's eyes. Inthis case, the image of the user's eyes captured by the imaging unit isanalyzed to detect the viewing direction, focal length, dilation of thepupils, the fundus pattern, and the movement of the eyelids.Alternatively, the sensor can be implemented by a light-emitting unitthat emits light to the eyes of the user and a light-receiving unit thatreceives light reflected from the eyes. For example, the thickness ofthe crystalline lens of the user can be detected from a received lightsignal.

The bio-sensor unit 11 outputs detection result information obtained bythese necessary sensors to the system controller 10.

The kinetic sensor unit 12 detects the user's kinetic information.Kinetic information includes, for example, information indicating theuser's state, such as a still, walking, or running state, informationindicating an exercising state (shaking, jumping, walking/runningrhythm, the center of gravity, etc.), or information indicating themovement of the user's body elements including the head, arms, legs,hands, and fingers.

These items of kinetic information can be detected using an accelerationsensor, a gyro (angular-velocity sensor), a vibration sensor, and thelike. That is, when an acceleration sensor or a gyro is provided, forexample, the movement of the entire body, head, neck, arms, and legs canbe detected as signals in accordance with the user's movement. Whendetecting the movement of arms or legs, an acceleration sensor or a gyromay be worn on the user's arms or legs.

The kinetic sensor unit 12 outputs detection result information obtainedby the acceleration sensor and the like to the system controller 10.

The position detecting unit 13 is, for example, a Global PositioningSystem (GPS) receiver. The GPS receiver receives radio waves from GPSsatellites and outputs latitude and longitude information indicating thecurrent position to the system controller 10.

Alternatively, the position detecting unit 13 may use wireless fidelity(WiFi) or a position information service provided by a cellular phonecompany.

The time-and-date counting unit 15 constantly performs a time-and-datecounting operation and counts the seconds, minutes, hours, days, months,and years.

The operation unit 16 is provided as an operation unit including keysand/or a dial used by the user of the information display apparatus 1 toenter various operations. Alternatively, the display unit 14 includes atouch panel, and this touch panel may be touched and operated as theoperation unit 16.

For example, the operation unit 16 may be operated to enter operationssuch as power on/off operations, display-related operations (e.g.,operations to select a display mode or perform display adjustment),various setting operations, and further, an operation to display a statehistory of the past, which is described later.

The system controller 10 performs a necessary control process based onoperation information from the operation unit 16.

Since the information display apparatus 1 in this example includes thebio-sensor unit 11 and the kinetic sensor unit 12, the user's intendedactions may be detected from bio- and kinetic information detected bythe bio-sensor unit 11 and the kinetic sensor unit 12, and the systemcontroller 10 may determine that the user's intended actions as enteredoperation information.

For example, when the user taps the user's fingers on the informationdisplay apparatus 1, the acceleration sensor or vibration sensor of thekinetic sensor unit 12 may detect this tapping, and the systemcontroller 10 may recognize this tapping as a user operation.

Alternatively, when the user rotates the head or shakes the neck, theacceleration sensor or angular-velocity sensor may detect this rotationor shaking, and the system controller 10 may recognize this rotation orshaking as a user operation.

Alternatively, the bio-sensor unit 11 may detect the movement of theuser's eyes, and the system controller 10 may recognize this movement(change in the viewing direction or blinking) which serves as the user'sintended action as a user operation.

The communication unit 18 performs data transmission/reception with anexternal apparatus. The communication unit 18 is only necessary to beconnected to a network via cable or wirelessly and to performcommunication. For example, the communication unit 18 may performcommunication over a network in a system shown in FIG. 9, which isdescribed later, or may directly communicate data with anotherapparatus.

Under control of the system controller 10, the storage unit 17 records(saves) various items of data and reproduces (reads) recorded data.

The storage unit 17 may be implemented by a fixed memory, such as a RAMor a flash memory, or may be implemented by, for example, a hard diskdrive (HDD).

Alternatively, the storage unit 17 may be implemented by, besides aninternal recording medium, a read/write drive corresponding to recordingmedia such as portable recording media, such as a memory card includinga fixed memory, an optical disk, a magneto-optical disk, and a hologrammemory.

Alternatively, the storage unit 17 may include both types, namely, aninternal-type memory such as a fixed memory or HDD and a read/writedrive corresponding to portable recording media.

In this example, the storage unit 17 includes storage areas serving as,more particularly, an avatar-information storage portion 17 a, adetected-information storage portion 17 b, a map database 17 c, and acommunication-data storage portion 17 d.

The avatar-information storage portion 17 a stores items of image dataserving as avatars to be displayed. For example, the avatar-informationstorage portion 17 a stores various items of image data indicating thedesign of each avatar and the expression and/or movement of each avatar.

For example, FIGS. 3A to 3F illustrate examples of avatars. FIGS. 3A, 3Band 3C illustrate exemplary images representing a still state, a walkingstate, and a running state, respectively, using movements andexpressions. FIG. 3D illustrates an exemplary image representing a statein which the avatar is still and has a fast pulse using a heart symbol.FIG. 3E illustrates an exemplary image representing a state in which theavatar is running and has a fast pulse using a heart symbol. FIG. 3Fillustrates an exemplary image representing a state in which the avataris depressed using an expression.

The avatar-information storage portion 17 a stores items of image datarepresenting, for example, the foregoing avatars and movements andexpressions of the avatars.

The detected-information storage portion 17 b stores bio-informationdetected by the bio-sensor unit 11, kinetic information detected by thekinetic sensor unit 12, and position information detected by theposition detecting unit 13. For example, under control of the systemcontroller 10, the detected-information storage portion 17 b storesthese items of information, together with time-and-date informationcounted by the time-and-date counting unit 15, at constant timeintervals. That is, the detected-information storage portion 17 b storesa history of the user's bio- and kinetic information and positioninformation.

The map database 17 c stores map images for displaying maps and othernecessary information.

The communication-data storage portion 17 d is used as a buffer orstorage of data transmitted from/received by the communication unit 18.

The display unit 14 includes a display panel portion implemented by, forexample, a liquid crystal panel or an organic electroluminescent (EL)panel and a display drive portion for driving the display panel portion.The display drive portion is implemented by a pixel drive circuit fordisplaying an image represented by supplied image data on the displaypanel portion. The pixel drive circuit applies drive signals based onvideo signals to pixels arranged in a matrix in the display panelportion at predetermined horizontal and vertical drive timings andcauses the pixels to display an image.

Under control of the system controller 10, the display unit 14 causesthe display panel portion to perform a predetermined display operation.More particularly in this example, the system controller 10 suppliesavatar data based on bio- and kinetic information to the display unit 14and causes the display unit 14 to display an avatar.

That is, the displayed avatar is illustrated in one of FIGS. 3A to 3F.The system controller 10 decides which avatar to display on the basis ofthe bio- and kinetic information and supplies the decided avatar to thedisplay unit 14.

In some cases, the system controller 10 may cause the display unit 14 todisplay a map image using map image data stored in the map database 17c.

It is preferable that the foregoing information display apparatus 1 beconstructed as a small and light-weight apparatus so that the user canwear the apparatus.

Depending on the details of bio-information to be detected, it ispreferable that the information display apparatus 1 be constructed as,for example, a wrist watch type, glasses type, headset type, hat type,helmet type, or glove type apparatus or as clothing including theapparatus. In particular, the information display apparatus 1 ispreferably constructed so that a part (the bio-sensor unit 11) of theinformation display apparatus 1 can be in contact with an appropriatebody part, such as the skin or head of a subject, in accordance withdetails of information to be detected.

Since it is only necessary that at least the bio-sensor unit 11 (or thekinetic sensor unit 12 depending on the details of kinetic informationto be detected) should be in contact with the user's body, theinformation display apparatus 1 may be constituted of two sections, asshown in FIG. 2.

FIG. 2 illustrates an example in which the information display apparatus1 is constructed as two separate sections, namely, a detection apparatussection 2 and a display apparatus section 3.

The detection apparatus section 2 includes the bio-sensor unit 11, thekinetic sensor unit 12, and a detected-information transmitter 19.

The display apparatus section 3 includes, besides the system controller10, the position detecting unit 13, the display unit 14, thetime-and-date counting unit 15, the operation unit 16, the storage unit17, and the communication unit 18, a detected-information receiver 20.

The detected-information receiver 20 and the detected-informationtransmitter 19 communicate with each other wirelessly or via cable. Whenwireless communication is performed, for example, a short distancewireless communication protocol, such as Bluetooth, may be employed.Alternatively, an optical communication protocol for performing datacommunication based on optical pulse modulation using visible light orinvisible light may be employed. Alternatively, a long distance wirelesscommunication protocol may be employed, or communication may beperformed over a network.

In the example shown in FIG. 2, bio-information detected by thebio-sensor unit 11 and kinetic information detected by the kineticsensor unit 12 are transmitted from the detected-information transmitter19 and received at the detected-information receiver 20.

The system controller 10 performs an avatar display control operationbased on the bio- and kinetic information received at thedetected-information receiver 20.

In the exemplary structure shown in FIG. 2, since it is only necessaryto make the detection apparatus section 2 come in contact with anecessary body portion such as the user's skin or head, the load on theuser to wear the detection apparatus section 2 can be alleviated. Inparticular, since the detection apparatus section 2 has a simplestructure including the bio-sensor unit 11, the kinetic sensor unit 12,and the detected-information transmitter 19, the size and weight of thedetection apparatus section 2 can be easily reduced. Accordingly, thedetection apparatus section 2 can be easily implemented as an apparatusthat can be worn on or at a necessary body part.

The display apparatus section 3 may be implemented as a small dedicatedapparatus carried by the user. Alternatively, for example, the displayapparatus section 3 may be implemented by adding the function of thedisplay apparatus section 3 to a portable apparatus, such as a cellularphone or a PDA. Alternatively, the display apparatus section 3 may notnecessarily be carried or worn by the user. In that case, the displayapparatus section 3 may be a relatively large apparatus. A desktop ornotebook personal computer may execute the function of the displayapparatus section 3.

1-2 Process of Displaying Avatar in Real-Time

Referring to FIGS. 4, 5A, and 5B, a process of displaying an avatarusing the information display apparatus 1 is described. The displayprocess described below is an operation of constantly detecting bio- andkinetic information and displaying an avatar on the display unit 14 inaccordance with the detected bio- and kinetic information, therebyrepresenting a user's state in real-time.

For example, the information display apparatus 1 performs an operationof displaying on the display unit 14 a user's state in an avatar displaymode started in response to an operation entered by the user.

FIG. 4 illustrates a control process performed by the system controller10 for displaying an avatar in real-time.

When the avatar display mode starts, the system controller 10 advancesthe process from step F101 to step F102 and starts a process ofdisplaying an avatar.

In step F102, the system controller 10 fetches bio- and kineticinformation as information detected by the bio-sensor unit 11 and thekinetic sensor unit 12. Further, the system controller 10 fetchesposition information detected by the position detecting unit 13.

Next in step F103, the system controller 10 stores the fetched bio- andkinetic information and position information, together with the currenttime and date (seconds, minutes, hours, days, months, and years) countedby the time-and-date counting unit 15 at that time, in thedetected-information storage portion 17 b of the storage unit 17.

Next in step F104, the system controller 10 determines the user's stateon the basis of the fetched bio- and kinetic information and decideswhich avatar to display.

The user's state may be a state represented by the bio- and kineticinformation itself or may be a psychological or emotional stateestimated from the bio-and kinetic information.

For example, whether the user's pulse is pounding can be determined fromthe heart rate or breathing information serving as bio-information.

Whether the user is still (standing or sitting), walking, or running canbe determined from kinetic information.

The user's psychological state can be determined on the basis ofbio-information. For example, the user's emotion (having fun, enjoying,or being happy, sad, afraid, calm, nostalgic, moved, surprised, excited,or stressed) can be estimated on the basis of a change in a numeralindicating bio-information due to a tense state, an excited state, or acomfortable state. This is because a detected value of the heart rate,pulse rate, brain wave, blood pressure, or GSR changes due to apsychological change.

Further, the user's state (calm, irritated, etc) can be determined fromthe state of the pupils or the movement of the eyes detected by a visionsensor.

Further, the movement of the user's body detected by the kinetic sensorunit 12 may be used to determine the user's state. For example, when theuser's pulse becomes fast, the user may be stressed or excited, or theuser may be exercising, such as running. In order to determine the causeof the user's state, reference may be made to information obtained froman acceleration sensor or the like.

It takes a certain amount of time to estimate the psychological state.In some cases, the psychological state may not be accurately estimatedsimply by temporarily fetching information detected by the bio-sensorunit 11. That is, information detected by the bio-sensor unit 11, whichserves as a signal obtained from a living body, changes its value everysecond. It is difficult to estimate the psychological state on the basisof the value obtained at a certain instance. In other cases, it isbetter to change a threshold for determining the physiological state inaccordance with continuous changes in emotion.

It is thus appropriate to determine the physiological state by referringto continuous changes in bio- and kinetic information from a certainpoint in the past. For example, the processing in steps F102 to F109 inFIG. 4 is repeated during a period in the avatar display mode. With theprocessing in step F103, bio-information at each point is stored in thedetected-information storage portion 17 b. In step F104, thepsychological state can be determined with reference not only tocurrently fetched bio-information, but also to items of storedbio-information from the past.

When the system controller 10 determines the user's state in, forexample, the foregoing manner, the system controller 10 selects anavatar that matches the user's state. That is, avatars in accordancewith the user's various states are stored in the avatar-informationstorage portion 17 a of the storage unit 17, as shown in FIGS. 3A to 3F.The system controller 10 selects, from among these avatars, an avatarthat matches the user's current state and decides that the selectedavatar is an avatar to be displayed.

For example, when the user's current state is determined as a state inwhich the user is running and the pulse rate is greater than or equal toa certain threshold, the system controller 10 selects an avatar shown inFIG. 3E. The avatar shown in FIG. 3E represents that, using itsexpression and movement, the user is running and the pulse rate is high.

When the user's current state is determined as a state in which the useris still and depressed, the system controller 10 selects an avatar shownin FIG. 3F, which represents such a state using an expression andmovement.

In step F105, the process is branched into two steps depending onwhether the current mode is a map display mode. The map display mode isa state in which a map is displayed on the display unit 14 in responseto a user operation. For example, when the user gives an instruction toselect the avatar display mode while a map is being displayed or whenthe user gives an instruction to display a map image during the avatardisplay mode while the process shown in FIG. 4 is being performed, it isdetermined in step F105 that the current mode is the map display mode.

When the current mode is not the map display mode, the system controller10 proceeds from step F105 to step F106 and performs an avatar displaycontrol operation. That is, the system controller 10 supplies the avatardata decided in step F104 to the display unit 14 and causes the displayunit 14 to display the avatar based on the avatar data.

In contrast, when the current mode is the map display mode, the systemcontroller 10 proceeds from step F105 to step F107 and, on the basis ofthe current position information, reads map image data in a range to bedisplayed from the map database 17 c. In step F108, the systemcontroller 10 generates display image data from the map image data andthe avatar data, supplies the generated display image data to thedisplay unit 14, and causes the display unit 14 to display the avatar ona map image. For example, the display unit 14 displays the avatarrepresenting the user at a position corresponding to the currentposition on the map image.

The process returns to step F102 and similar processing is repeateduntil it is determined in step F109 that the avatar display mode isterminated.

For example, when the user enters a user operation to give aninstruction to terminate the avatar display mode, the system controller10 proceeds from step F109 to step F110 and terminates displaying theavatar on the display unit 14. Accordingly, the process shown in FIG. 4is terminated.

FIGS. 5A and 5B illustrate examples of graphical representations.

FIG. 5A illustrates an example of a graphical representation displayedin response to the processing in step F106. In this case, an avatar 30is displayed in the corner of a screen 14 a of the display unit 14. Forexample, when a necessary image is displayed in the center of the screen14 a in a normal state, the avatar 30 is simply displayed in the cornerof the screen 14 a in a continuous manner. Alternatively, the avatar 30may be enlarged and displayed in the center of the screen 14 a.

Since the process shown in FIG. 4 is continuously performed during theavatar display mode, the displayed avatar 30 changes according to theuser's state. For example, when the user starts running, an image shownin FIG. 3C is displayed. When the user continues running and has afaster pulse, the avatar 30 changes to an image shown in FIG. 3E.

FIG. 5B illustrates an example of a graphical representation displayedin response to the processing in step F108. A map image 31 is displayedon the screen 14 a. In addition, the avatar 30 is displayed while theuser's position (position based on position information detected by theposition detecting unit 13) is indicated on the map. In this case, sincethe process shown in FIG. 4 is continuously performed, as the user'sposition moves or the user's state changes, the range of the displayedmap image, the user's current position on the map, and the avatar 30change as well.

By displaying an avatar in the foregoing manner, the user's state can berepresented using an interesting image, which becomes more enjoyable tothe user.

When the avatar is displayed together with the map image, the user'sposition and state can be clearly represented additionally using, forexample, a navigation function.

Although the exemplary avatars in accordance with the user's variousstates are illustrated in FIGS. 3A to 3F, they are only examples inaccordance with some of the user's states, and images representing theuser's more versatile states can be prepared. For example, avatarshaving expressions and movements that represent various states, such asstates in which the user feels hot or cold, the user is comfortable,tired, sleepy, or excited, the user has a low or high blood pressure,the user has raspy or easy breathing, or the user has a fast or slowheart rate, are prepared. It is only necessary to select one from amongthese avatars and display the selected avatar.

Avatars representing a certain user's state may be still images, movingimages, or pseudo-moving images. Pseudo-moving images are in a displayformat in which a few (two or three) still images are alternatelydisplayed.

Specific avatars may be prepared and stored in advance in theavatar-information storage portion 17 a.

Alternatively, for example, the communication unit 18 may communicatewith an external server (e.g., the server apparatus 70 described laterwith reference to FIGS. 9 and 11) and download avatar data.

Alternatively, many basic avatars may be prepared and stored in advance,and a user may select one from among these basic avatars. Alternatively,a user may be allowed

1-3 Process of Reproducing and Displaying Avatars as State History

By performing the processing in step F103 in FIG. 4, bio-information,kinetic information, and position information are sequentially stored inthe detected-information storage portion 17 b.

Even in a period outside the avatar display mode (period in which theprocess shown in FIG. 4 is not performed), when the system controller 10performs the processing in steps F102 and F103 at, for example, constanttime intervals, the user's bio-information, kinetic information, andposition information are constantly stored in the detected-informationstorage portion 17 b.

FIG. 6A illustrates an example of information stored in thedetected-information storage portion 17 b.

Referring to FIG. 6A, position information PL (PL1, PL2, PL3, . . . ),bio-information L (L1, L2, L3, . . . ), and kinetic information M (M1,M2, M3, . . . ) detected at corresponding times and dates are stored inassociation with corresponding items of time-and-date information.

For example, since detected items of bio-information, kineticinformation, and position information are successively stored togetherwith items of time-and-date information in the avatar-informationstorage portion 17 a, the user's states in the past can be determined.

In step F104 of FIG. 4, the user's current state is determined, andwhich avatar to display is decided. Information indicating the selectedavatar (or information on the user's state based on which the avatar isselected) may be stored as avatar-related information C (C1, C2, C3, . .. ) shown in FIG. 6B.

Further, avatar-related information C and position information PL may bestored in association with time-and-date information, but nobio-information or kinetic information may be stored, which is not shownin FIGS. 6A and 6B.

By storing these items of information, the information display apparatus1 can reproduce and display avatars representing the user's states inthe past.

FIG. 7 illustrates a process performed by the system controller 10 whenreproducing and displaying avatars representing a user's state history.

When the user operates the operation unit 16 to give an instruction toreproduce an avatar, the system controller 10 proceeds from step F201 tostep F202 and displays an image on the display unit 14 to prompt theuser to enter a reproducing criterion.

A reproducing criterion, such as time and date, place, or the like, is acriterion used to search for stored data to reproduce. For example, whena time and date is used as a reproducing criterion, the user may beallowed to specify a specific date as X(month)/Y(day), a specific timeand date as X(month)/Y(day), Z PM, or a specific range from A (month) /B(day) to X(month)/Y(day).

When a place is used as a reproducing criterion, the user may be allowedto specify the name of a place or to specify the range of a region on amap image.

Alternatively, the user may be allowed to specify bio-information,kinetic information, psychological state, or the like as a reproducingcriterion. For example, the user may be allowed to specify that “theheart rate is greater than or equal to XX”, “the user is running”, or“the user is depressed”.

The user may further add an AND operator or an OR operator toreproducing criteria, such as time and date, place, bio-information,kinetic information, and psychological state.

Alternatively, the user may specify to search for “all stored data”.

The system controller 10 displays an image so that such a reproducingcriterion described above can be entered and, in steps F203 and F204,waits for the user to enter a reproducing criterion.

When the user enters a reproducing cancel operation, the flow proceedsfrom step F204 to F209, and the system controller 10 stops displayingthe image for prompting the user to enter a reproducing criterion. Theprocess shown in FIG. 7 is terminated.

When the user enters a reproducing criterion, the flow proceeds fromstep F203 to step F205, and the system controller 10 extracts, fromamong items of data stored in the detected-information storage portion17 b, data matching the reproducing criterion and regards the extracteddata as data to be reproduced.

For example, when a specific time and date is specified, a search basedon the entered time and date is conducted through items of data storedas shown in FIG. 6A or 6B to extract data matching the entered time anddate. Having extracted the data to be reproduced, in step F206, thesystem controller 10 controls reproducing the extracted data.

For example, FIGS. 8A and 8B show cases in which items of data shown inFIGS. 6A and 6B are reproduced and images represented by the reproduceditems of data are displayed.

FIG. 8A shows an example in which a list of images represented byextracted items of data is displayed on the screen 14 a of the displayunit 14. In this exemplary case, a time-and-date indication 32 based ontime-and-date information and a name-of-place indication 33 based onposition information PL are displayed together with the avatar 30.

Alternatively, as shown in FIG. 8B, images represented by extracteditems of data may be displayed on a map image. In this case, while aposition based on position information PL is indicated on the map image31, the corresponding avatar 30 and the time and date are displayed.

Other various display formats are conceivable.

When items of data are stored in the format shown in FIG. 6A, it isnecessary to perform, for data determined to be reproduced, a process ofdetermining the user's state on the basis of bio- and kineticinformation and deciding which avatar to display (process similar tostep F104 shown in FIG. 4). In contrast, when items of avatar-relatedinformation C are stored as shown in FIG. 6B, it is only necessary todisplay avatars represented by the avatar-related information C, asshown in FIGS. 8A and 8B.

In the graphical representations shown in FIGS. 8A and 8B, the displayeddetails are changed as the user scrolls on the screen or skips a page.

Alternatively, stored data to be reproduced may be automaticallychanged.

When the system controller 10 determines that reproducing is terminatedin response to a user operation or a certain condition, the processproceeds from step F207 to step F208, and the system controller 10terminates the reproducing and displaying process. The process shown inFIG. 7 is terminated.

For example, when avatars are reproduced and displayed on the basis ofitems of data stored in the detected-information storage portion 17 b,the user can view the user's states or the places where the user hasbeen at arbitrary points in the past. Accordingly, the user can easilyremember and identify the places where the user has been and the user'sstates at those points in the past, which serve as the user's statehistory.

2. Displaying Avatar Based on Another Person's Bio- and KineticInformation 2-1 System Configuration

The foregoing description concerns the operation of the informationdisplay apparatus 1 which displays an avatar in accordance with theuser's current state or which reproduces and displays avatarsrepresenting the user's past states. Hereinafter, an informationprocessing system in which the user can be informed of another person'sstate using an avatar and apparatuses constituting the system aredescribed.

FIG. 9 illustrates an example of a system configuration. Thisinformation processing system is an exemplary system in whichtransmitting apparatuses 100 (hereinafter may be collectively referredto as a “transmitting apparatus 100” when it is unnecessary todistinguish between the two), the server apparatus 70, and a displayapparatus 200 can communicate with one another via the network 60.

In this system, a user of the display apparatus 200 can be informed ofthe state of another user wearing the transmitting apparatus 100 usingan avatar.

The network 60 includes various types, such as the Internet, a mobilephone communication network, a personal handy phone (PHS) communicationnetwork, an ad-hoc network, and a local area network (LAN).

The transmitting apparatus 100 detects bio-information, kineticinformation, position information, and time-and-date information of aperson wearing the transmitting apparatus 100 and transmits the detecteditems of information to the server apparatus 70 by communicating via thenetwork 60. For example, the transmitting apparatus 100 periodicallytransmits information.

The server apparatus 70 stores the transmitted bio-information, kineticinformation, position information, and time-and-date information in aninternal database. These items of information are stored in associationwith identification information (hereinafter referred to as a “user ID”)uniquely given to the person wearing the transmitting apparatus 100,which will be described later.

The display apparatus 200 can obtain bio-information, kineticinformation, position information, and time-and-date information of aspecific user (another person), which are stored in the server apparatus70, by communicating via the network 60. On the basis of the obtaineditems of information including bio-and kinetic information and the userID, the display apparatus 200 decides which avatar to display anddisplays the avatar. When displaying the avatar, the display apparatus200 may cause the avatar to reflect the position information and thetime-and-date information.

Alternatively, the display apparatus 200 may obtain an avatar itself,which is to be displayed, from the server apparatus 70 by communicatingvia the network 60 and display the avatar.

The information display apparatus 1 with the structure shown in FIG. 1may serve as the transmitting apparatus 100 or the display apparatus 200shown in FIG. 9.

2-2 Structure of Transmitting Apparatus

FIG. 10 shows an exemplary structure of the transmitting apparatus 100shown in FIG. 9.

The transmitting apparatus 100 includes a detection/transmissioncontroller 110, a bio-sensor unit 111, a kinetic sensor unit 112, aposition detecting unit 113, a time-and-date counting unit 115, anoperation unit 116, a storage unit 117, and a communication unit 118.

The detection/transmission controller 110 is implemented by, forexample, a CPU. The detection/transmission controller 110 performs acontrol process for detecting and transmitting bio-information.

The storage unit 117 has storage areas including a ROM, a RAM, and anon-volatile memory and is used as a storage area for a processingprogram executed by the detection/transmission controller 110 and a workarea. Alternatively, the storage unit 117 may be implemented by aninternal memory of a microchip computer serving as thedetection/transmission controller 110.

The non-volatile memory area in the storage unit 117 may storeidentification information (apparatus ID) uniquely given to each of thetransmitting apparatuses 100 and identification information (user ID) ofthe user of each of the transmitting apparatuses 100.

The communication unit 118 performs data transmission/reception with anexternal apparatus. In particular, in the case of the systemconfiguration shown in FIG. 9, the communication unit 118 performs datacommunication with the server apparatus 70 as communication via thenetwork 60. In this case, the communication unit 118 is only necessaryto be connected to the network 60 via cable or wirelessly and to performcommunication. For example, the communication unit 118 may performwireless communication with a network access point.

The operation unit 116 is provided for the user to enter operationsnecessary for using the transmitting apparatus 100. Using the operationunit 116, the user may enter, for example, power on/off operations andvarious setting operations.

The bio-sensor unit 111, the kinetic sensor unit 112, the positiondetecting unit 113, and the time-and-date counting unit 115 havefunctions similar to those of the bio-sensor unit 11, the kinetic sensorunit 12, the position detecting unit 13, and the time-and-date countingunit 15 of the information display apparatus 1 illustrated in FIG. 1.With these units, the user's bio-information, kinetic information,position information, and time-and-date information are detected.

In the transmitting apparatus 100 described above, thedetection/transmission controller 110 periodically stores, for example,the user's bio-information, kinetic information, position information,and time-and-date information at a detection point, which are detectedby the bio-sensor unit 111, the kinetic sensor unit 112, the positiondetecting unit 113, and the time-and-date counting unit 115,respectively, in the storage unit 117. Using the bio-information,kinetic information, position information, and time-and-date informationfetched into the storage unit 117, the detection/transmission controller110 generates transmission data and causes the communication unit 118 totransmit the transmission data to the server apparatus 70 via thenetwork 60. In this case, the transmission data includes, besides thebio-information, kinetic information, position information, andtime-and-date information, the user ID or the apparatus ID.

It is preferable that the transmitting apparatus 100 described above beconstructed as a small and light-weight apparatus so that the user caneasily wear the apparatus. Depending on the details of bio- and kineticinformation to be detected, it is preferable that the transmittingapparatus 100 be constructed as, for example, a wrist watch type,glasses type, headset type, hat type, helmet type, or glove typeapparatus or as clothing including the apparatus. In particular, thetransmitting apparatus 100 is preferably constructed so that a part (thebio-sensor unit 111) of the transmitting apparatus 100 can be in contactwith an appropriate body part, such as the skin or head of a subject, inaccordance with details of information to be detected.

Alternatively, as in the example shown in FIG. 2, the bio-sensor unit111 and the kinetic sensor unit 112 may be constructed as independentunits, which are not shown in a drawing.

As is clear from the comparison of the structure shown in FIG. 10 withthe structures shown in FIGS. 1 and 2, the information display apparatus1 illustrated in FIGS. 1 and 2 may perform the same operation as thetransmitting apparatus 100 shown in FIG. 10 and function as thetransmitting apparatus 100 shown in FIG. 9. That is, for example, thesystem controller 10 shown in FIG. 1 may periodically generatetransmission data using bio-information, kinetic information, positioninformation, and time-and-date information and cause the communicationunit 18 to transmit the transmission data to the server apparatus 70 viathe network 60.

2-3 Structure of Server Apparatus

FIG. 11 shows an exemplary structure of the server apparatus 70.

As has been described above, the server apparatus 70 is an apparatusthat can store, for example, bio-information, kinetic information,position information, and time-and-date information transmitted from thetransmitting apparatus 100 by communicating via the network 60 andtransmit the stored items of information to the display apparatus 200.

The server apparatus 70 includes a server controller 72, a networkstorage unit 71, a communication unit 73, an information management unit74, and a bio-and-kinetic-information database 75. The server apparatus70 may further include an avatar-information storage unit 76 and a mapdatabase 77.

The server controller 72 performs operation control necessary as theserver apparatus 70. In particular, the server controller 72 controlsthe network communication operation, processes performed upon receipt ofbio-information, kinetic information, position information, andtime-and-date information transmitted from the transmitting apparatus100, and transmission of these items of information to the displayapparatus 200.

The network storage unit 71 is implemented by, for example, an HDD. Forexample, the network storage unit 71 temporarily storestransmission/reception data communicated between the transmittingapparatus 100 and the display apparatus 200 via the network 60 andstores various items of necessary data.

The communication unit 73 performs data communication via the network 60with the transmitting apparatus 100 and the display apparatus 200.

The information management unit 74 manages the bio-information, kineticinformation, position information, and time-and-date informationtransmitted from the transmitting apparatus 100.

The bio-and-kinetic-information database 75 stores the bio-information,kinetic information, position information, and time-and-date informationtransmitted from the transmitting apparatus 100 in association with, forexample, the user ID in the bio-and-kinetic-information database 75.

In addition, the information management unit 74 registers data into thebio-and-kinetic-information database 75 and searches for data in thebio-and-kinetic-information database 75.

The avatar-information storage unit 76 stores various avatars to bedisplayed using the display apparatus 200. In particular, theavatar-information storage unit 76 may store avatars so that each avataris managed in association with a user ID.

The map database 77 stores map images for displaying maps using thedisplay apparatus 200 and other necessary data.

2-4 Structure of Display Apparatus

Referring now to FIGS. 12 and 13, examples of the structure of thedisplay apparatus 200 will be described.

The display apparatus 200 shown in FIG. 12 includes a system controller201, a communication unit 218, a display unit 214, an operation unit216, and a storage unit 217. A storage area serving as acommunication-data storage portion 217 d is provided in the storage unit217.

The example of the structure of the display apparatus 200 shown in FIG.13 includes, in addition to the structure shown in FIG. 12, storageareas serving as an avatar-information storage portion 217 a and a mapdatabase 217 c provided in the storage unit 217.

Since these units of the structure shown in each of FIGS. 12 and 13 aresimilar to the system controller 10, the communication unit 18, thedisplay unit 14, the operation unit 16, and the storage unit 17 shown inFIG. 1, descriptions thereof are not repeated to avoid redundancy.

In particular, the communication unit 218 performs data communicationwith the server apparatus 70 as communication via the network 60 shownin FIG. 9.

The system controller 201 controls an avatar display operation and aprocess of communicating with the server apparatus 70.

The display apparatus 200 illustrated in the system configuration shownin FIG. 9 is only necessary to display, on the display unit 214, anavatar representing the state of a certain person (the user of thetransmitting apparatus 100, who is a stranger to the user of the displayapparatus 200) on the basis of information received from the serverapparatus 70.

Accordingly, two types of the structure of the display apparatus 200 areconceivable in view of the system operation.

An avatar in this example is an image representing a user's statedetermined on the basis of bio- and kinetic information.

That is, which avatar to display is decided on the basis of the user'sstate determined from bio- and kinetic information. In view of thesystem operation, a process of deciding which avatar to display may beperformed using the server apparatus 70 or the display apparatus 200.

When the server apparatus 70 decides which avatar to display on thebasis of the user's state determined from bio- and kinetic information,the server apparatus 70 can transmit avatar data itself to the displayapparatus 200.

In this case, the display apparatus 200 stores the received avatar datain the communication-data storage portion 217 d and thereafter displaysthe avatar using the avatar data. Accordingly, the display apparatus 200can be realized as the structure shown in FIG. 12.

Alternatively, when the server apparatus 70 decides which avatar todisplay on the basis of the user's state determined from bio- andkinetic information, the server apparatus 70 may transmit, instead ofavatar data itself, information for specifying an avatar to be displayedor information for specifying the expression or movement of an avatar tobe displayed to the display apparatus 200.

In this case, the display apparatus 200 selects an avatar on the basisof the received information and displays the avatar. Accordingly, it ispreferable for the display apparatus 200 to have the structure shown inFIG. 13, which includes the avatar-information storage portion 217 a.

When displaying a map image, in the case where the display apparatus 200downloads map image data from the server apparatus 70, the displayapparatus 200 stores the downloaded map data in the communication-datastorage portion 217 d and thereafter displays the map image using themap data. Accordingly, the display apparatus 200 can have the structureshown in FIG. 12, which has no map database.

In contrast, when the system operation involves the display apparatus200 performing a process of deciding which avatar to display, thestructure shown in FIG. 13 is appropriate.

In this case, the server apparatus 70 at least transmits a certainperson's (user ID) bio- and kinetic information to the display apparatus200, and the display apparatus 200 decides which avatar to display onthe basis of the user's state determined from the received bio- andkinetic information and displays the avatar. Accordingly, the serverapparatus 70 is necessary to have the avatar-information storage portion217 a.

As is clear from the foregoing description, the structure shown in FIG.12 corresponds to the case which assumes the system operation in whichthe server apparatus 70 transmits avatar data and which intends tosimplify the display apparatus 200. In contrast, the structure shown inFIG. 13 is the exemplary structure that can be used regardless of whichof the server apparatus 70 and the display apparatus 200 performs theavatar deciding process.

Similarly, the information display apparatus 1 shown in FIG. 1 mayfunction as the display apparatus 200 regardless of which of the serverapparatus 70 and the information display apparatus 1 performs the avatardeciding process.

2-5 System Operation for Displaying Avatar

An exemplary operation of the information processing system includingthe transmitting apparatus 100, the server apparatus 70, and the displayapparatus 200 is described.

FIG. 14 illustrates the operation of transmitting bio-information andthe like from the transmitting apparatus 100 to the server apparatus 70.Referring to FIG. 14, a process performed by the transmitting apparatus100 is a process executed under control of the detection/transmissioncontroller 110, and a process performed by the server apparatus 70 is aprocess executed under control of the server controller 72.

In the transmitting apparatus 100, the detection/transmission controller110 waits for a transmission timing in step F301. When the transmissiontiming has come, the flow proceeds to step F302. The transmission timingmay be, for example, a periodical timing or a timing based on a useroperation or any other trigger.

In step F302, the detection/transmission controller 110 fetchesbio-information obtained by the bio-sensor unit 111, kinetic informationobtained by the kinetic sensor unit 112, position information obtainedby the position detecting unit 113, and the current time-and-dateinformation obtained by the time-and-date counting unit 115 and storesthese items of information in the storage unit 117.

In step F303, the detection/transmission controller 110 generatestransmission data. That is, the detection/transmission controller 110generates transmission data including the bio-information, kineticinformation, position-information, and time-and-date information fetchedinto the storage unit 117, and the user ID or the apparatus ID.

In step F304, the transmitting apparatus 100 establishes a communicationconnection with the server apparatus 70. The detection/transmissioncontroller 110 causes the communication unit 118 to start communicatingvia the network 60 to establish a communication connection with theserver apparatus 70. In this case, in step F401, the server controller72 of the server apparatus 70 causes the communication unit 73 toperform a communication connection process and performs anauthentication process. The authentication process can be performedusing various methods. For example, in one method, the transmittingapparatus 100 transmits its apparatus ID to the server apparatus 70, andthe server apparatus 70 determines whether this apparatus ID is anappropriately registered apparatus ID.

When the authentication is successful and a connection is established,the transmitting apparatus 100 transmits data. That is, thedetection/transmission controller 110 causes the communication unit 118to transmit data including bio-information, kinetic information,position information, time-and-date information, and user ID (orapparatus ID).

In contrast, the server controller 72 of the server apparatus 70fetches, in step F402, the data received at the communication unit 73into the network storage unit 71.

When the reception data is completely fetched, in step F403, the servercontroller 72 decodes the fetched reception data and extracts data fromthe decoded data. The server controller 72 transfers bio-information,kinetic information, position information, time-and-date information,and user ID (or apparatus ID) included in the reception data to theinformation management unit 74 and registers these items of data in thebio-and-kinetic-information database 75.

Until the operation is terminated (such as by turning off the power),the transmitting apparatus 100 returns from step F306 to step F301 and,for example, every time the periodical transmission timing has come,performs the processing in steps F302 to F305 described above.

For example, when the foregoing process is performed in a sequentialmanner, the bio-and-kinetic-information database 75 of the serverapparatus 70 stores information transmitted from the transmittingapparatus 100.

FIG. 15 illustrates an example of a registration format of thebio-and-kinetic-information database 75.

In the bio-and-kinetic-information database 75, for example,bio-information L, kinetic information M, position information PL, andtime-and-date information Date are stored in association with the userID (UID 1, UID 2, . . . ).

For example, every time bio-information L, kinetic information M,position information PL, and time-and-date information Date aretransmitted, together with the user ID (UID 1), from the transmittingapparatus 100 worn by the user whose user ID is UID 1 by performing theprocess shown in FIG. 14, the bio-information L, kinetic information M,position information PL, and time-and-date information Date areadditionally stored in association with the user ID (UID 1), as shown inFIG. 15, as items of data in one registration unit.

Regarding the stored data, the number of registration units stored foreach user ID may be limited, and older data may be deleted one afteranother. Alternatively, older data may be deleted with reference to thetime-and-date information.

Alternatively, data (L, M, PL, and Date) in only one registration unitmay be stored for each user ID. That is, the storage format may be suchthat, every time bio-information L, kinetic information M, positioninformation PL, and time-and-date information Date for a certain user IDare transmitted, the bio-information L, kinetic information M, positioninformation PL, and time-and-date information Date for that user ID areupdated.

Alternatively, a database registering bio-information and the likeusing, instead of the user ID (or together with the user ID), theapparatus ID may be employed.

On the basis of these items of information stored in the serverapparatus 70 in this manner, the display apparatus 200 can display anavatar representing the state of a person who is a stranger to the userof the display apparatus 200.

Examples of processes for performing the foregoing operations areillustrates in FIGS. 16 and 17.

FIGS. 16 and 17 illustrate processes performed by the server apparatus70 and the display apparatus 200. A process performed by the serverapparatus 70 is a process under control of the server controller 72. Aprocess performed by the display apparatus 200 is a process undercontrol of the system controller 201.

FIG. 16 illustrates an exemplary process performed in the case where theserver apparatus 70 decides on an avatar. FIG. 17 illustrates anotherexemplary process performed in the case where the display apparatus 200decides on an avatar.

The exemplary process shown in FIG. 16 is described.

When the user of the display apparatus 200 wants to be informed of thecurrent state of another person (such as a friend), the user enters anoperation to specify that other person's user ID (or apparatus ID) andinstructs the display apparatus 200 to access the server apparatus 70.For example, when the user knows the other person's user ID, the usercan enter an operation to request for a graphical representation of theother person's state by specifying the other person's user ID.

In response to such a user operation, the display apparatus 200 performsa process shown in FIG. 16 in order to display an avatar representingthe other person's state.

In step F601, the display apparatus 200 establishes a communicationconnection with the server apparatus 70. The system controller 201causes the communication unit 218 to start communicating via the network60 to establish a communication connection with the server apparatus 70.In this case, in step F501, the server controller 72 of the serverapparatus 70 causes the communication unit 73 to perform a communicationconnection process and performs an authentication process. Also in thiscase, the authentication process can be performed using various methods.For example, in one method, the display apparatus 200 transmits the userID of the user thereof or the apparatus ID thereof to the serverapparatus 70, and the server apparatus 70 checks the user ID or theapparatus ID.

When the authentication of communication with the server apparatus 70 issuccessful and a connection with the server apparatus 70 is established,the system controller 201 of the display apparatus 200 transmits, instep F602, a request for data, which includes the other person's user ID(or apparatus ID) specified by the user of the display apparatus 200, tothe server apparatus 70.

Upon detection of receipt of the request for the data in step F502, theserver controller 72 transfers in step F503 the user ID (or apparatusID) to the information management unit 74 and gives an instruction tosearch for the corresponding data. The information management unit 74extracts, from the bio-and-kinetic-information database 75, data(including bio-information L, kinetic information M, positioninformation PL, and time-and-date information Date) corresponding to thespecified user ID (or apparatus ID). In this case, when bio-informationL, kinetic information M, position information PL, and time-and-dateinformation Date constitute one registration data unit and when multipleregistration data units are stored, bio-information L, kineticinformation M, position information PL, and time-and-date informationDate serving as the most recent registration data unit may be read onthe basis of the time-and-date information Date.

Next in step F504, the server controller 72 causes the informationmanagement unit 74 to perform a process of deciding which avatar todisplay.

On the basis of the read bio-information L, kinetic information M,position information PL, and time-and-date information Date, theinformation management unit 74 determines the user's state particularlyusing the bio-information L and the kinetic information M. From amongvarious avatars stored in the avatar-information storage unit 76, theinformation management unit 74 decides on an avatar with an expressionor movement in accordance with the determined user's state.

In this case, when different avatars are prepared for different userIDs, the user ID is additionally used to decide which avatar to display.

In step F505, the server controller 72 instructs the informationmanagement unit 74 to read map image data. On the basis of positioninformation PL read from the bio-and-kinetic-information database 75,the information management unit 74 reads, from the map database 77, mapimage data within a necessary range including a position indicated bythe position information PL.

In step F506, the server controller 72 receives, from the informationmanagement unit 74, data of the decided avatar to be displayed (e.g.,avatar data itself), the position information PL, the map image data,and the time-and-date information Date and causes the communication unit73 to transmit these items of information to the display apparatus 200.

The data of the avatar appropriately includes, when the displayapparatus 200 has the structure shown in FIG. 12, the avatar dataitself. However, when the display apparatus 200 stores avatars as in thestructure shown in FIG. 13 or FIG. 1, the data of the avatar may includeinformation for specifying the avatar or information for determining theuser's state in order to specify the avatar.

In step F603, the system controller 201 of the display apparatus 200receives the data of the avatar, the map image data, the positioninformation PL, and the time-and-date information Date. That is, theseitems of data received at the communication unit 218 are stored in thecommunication-data storage portion 217 d of the storage unit 217.

On the basis of the received and saved data, the avatar is displayed.

In this case, the system controller 201 determines, in step F604,whether the current mode is the map display mode. This is similar to theprocessing in step F105 of FIG. 4.

When the current mode is not the map display mode, in step F605, thesystem controller 201 causes the display unit 214 to display the avatar.That is, the system controller 201 supplies the received and savedavatar data to the display unit 214 and causes the display unit 214 todisplay the avatar illustrated in, for example, FIG. 5A.

In contrast, when the current mode is the map display mode, in stepF606, the system controller 201 causes the display unit 214 to displaythe avatar on a map image. That is, the system controller 201 suppliesthe received and saved avatar data and map image data to the displayunit 214 and causes the display unit 214 to display the avatar and themap illustrated in, for example, FIG. 5B.

In this case, from the avatar, the user of the display apparatus 200 canbe informed of the current state (state based on the most recent bio-and kinetic information stored in the server apparatus 70) of the otherperson (friend).

FIG. 17 illustrates an exemplary process performed in the case where thedisplay apparatus 200 decides which avatar to display. Steps in FIG. 17which are the same as those in FIG. 16 are given the same step numerals.

Steps F601 and F602 of the display apparatus 200 and steps F501 and F502of the server apparatus 70 are the same as those shown in FIG. 16. Withthe processing in these steps, communication is established, and arequest for data is transmitted and received.

Upon detection of receipt of the request for the data in step F502, theserver controller 72 transfers in step F503 the user ID (or apparatusID) to the information management unit 74 and gives an instruction tosearch for the corresponding data. In response to this, as in FIG. 16,the information management unit 74 extracts, from thebio-and-kinetic-information database 75, data (including bio-informationL, kinetic information M, position information PL, and time-and-dateinformation Date) corresponding to the specified user ID (or apparatusID).

In step F510, the server controller 72 receives the bio-information L,kinetic information M, position information PL, and time-and-dateinformation Date read by the information management unit 74 and causesthe communication unit 73 to transmit these items of information to thedisplay apparatus 200.

In step F610, the system controller 201 of the display apparatus 200receives the bio-information L, kinetic information M, positioninformation PL, and time-and-date information Date. That is, these itemsof data received at the communication unit 218 are stored in thecommunication-data storage portion 217 d of the storage unit 217.

Next in step F611, the system controller 201 performs a process ofdeciding which avatar to display. That is, using the receivedbio-information L and kinetic information M, the system controller 201determines the other person's user state. From among various avatarsstored in the avatar-information storage portion 17 a, the systemcontroller 201 decides on an avatar with an expression or movement inaccordance with the determined user's state.

In this case, when different avatars are prepared for different userIDs, the other person's user ID specified at the time access is gainedto the server apparatus 70 is additionally used to decide which avatarto display.

When which avatar to display is decided, the avatar is displayed.

In this case, the system controller 201 determines, in step F612,whether the current mode is the map display mode. This is similar to theprocessing in step F105 of FIG. 4.

When the current mode is not the map display mode, in step F613, thesystem controller 201 causes the display unit 214 to display the avatar.That is, the system controller 201 supplies data of the decided avatarto the display unit 214 and causes the display unit 214 to display theavatar illustrated in, for example, FIG. 5A.

In contrast, when the current mode is the map display mode, in stepF614, the system controller 10 reads from the map database 17 c mapimage data within a range to be displayed on the basis of the receivedposition information PL and causes the display unit 214 to display theavatar on a map image. That is, the system controller 201 supplies theavatar data and the map image data to the display unit 214 and causesthe display unit 214 to display the avatar and the map image illustratedin, for example, FIG. 5B.

In this case, from the avatar, the user of the display apparatus 200 canbe informed of the current state (state based on the most recent bio-and kinetic information stored in the server apparatus 70) of the otherperson (friend).

When the foregoing system operation is performed, the user of thedisplay apparatus 200 can be informed, from a displayed avatar, thecurrent state of another person who is wearing the transmittingapparatus 100. For example, the user of the display apparatus 200 cancheck, from avatars, various states of a friend, such as states in whichthe friend is running, depressed, and the like.

In the foregoing description, the current state of another person isdisplayed using an avatar. Alternatively, the past state of anotherperson may be displayed.

For example, in the case where bio-information and the like of the userof the transmitting apparatus 100 are stored in thebio-and-kinetic-information database 75 of the server apparatus 70 for acertain period of time, when a point in the past within this informationstorage range is specified and when this other person's bio-informationand the like can be extracted from the stored information, the paststate of this other person can be displayed using an avatar.

In this example, the user of the display apparatus 200 is informed ofthe state of another person, such as a friend, provided that the user ofthe display apparatus 200 knows the user ID of this other person.Alternatively, the user of the display apparatus 200 may be informed ofthe state of a complete stranger or a celebrity using a displayedavatar. For example, when a certain person authorizes to make his or herown state public and registers so in the system, the user of the displayapparatus 200 can arbitrary display the state of this person who hasauthorized to make his or her own state public.

3. Displaying Avatars Based on User's and Another Person's Bio- andKinetic Information

The foregoing description concerns the example in which the informationdisplay apparatus 1 displays an avatar representing the state of theuser of the information display apparatus 1 and the example in which thedisplay apparatus 200 displays an avatar representing another person'sstate. Alternatively, both the user's state and another person's statecan be simultaneously displayed using avatars.

For example, in the system configuration shown in FIG. 9, it is assumedthat the display apparatus 200 has the structure serving as theinformation display apparatus 1 shown in FIG. 1.

The information display apparatus 1 may perform the process shown inFIG. 4 in order to display an avatar representing the state of the userof the information display apparatus 1. In order to display an avatarrepresenting another user's state, the information display apparatus 1may perform the operation described as the process performed by thedisplay apparatus 200 in FIG. 16 or 17.

That is, in the structure shown in FIG. 1, when the system controller 10performs in parallel the process shown in FIG. 4 and the process shownin FIG. 16 or 17, the display unit 14 can display avatars representingthe user and the other person.

For example, two avatars may be displayed on a screen. Alternatively,referring to FIG. 18, the user's state and another person's state may berepresented using avatars 30A and 30B, respectively, while the positionsof the user and the other person may be indicated on the map image 31.

In the example shown in FIG. 18, for example, the state of the user ofthe information display apparatus 1, who is waiting for another personat the place of appointment, is represented by indicating the positionof the user on the map image 31 and displaying the avatar 30Brepresenting the user's state. In addition, the state of the otherperson who is late for the appointment and is thus running is displayedusing the avatar 30A at the current position.

4. Advantages and Modifications of Embodiment

According to the above-described embodiment, the user of the informationdisplay apparatus 1 (or the display apparatus 200) can recognize thestate of the user or another person from the expression or movement ofan avatar. Accordingly, the user can be informed of the state of theuser or another person with ease and delight.

For example, when an avatar representing the user's state is displayed,the user can enjoy the display screen or be informed of the precisestate, such as tired or depressed, so that the user can pay attention tothe user's subsequent behavior.

When the user's past state is displayed using an avatar, the user canprecisely remember the user's behavior and emotions and can enjoy theuser's memory.

When another person's state is displayed using an avatar, the user canbe informed of the precise state of the other person.

For example, the user can be informed of the state and the currentposition of the user's friend who is late for an appointment with theuser. In addition, the parents of a child can be informed of the currentplace and position where the child is. Further, the state of anunhealthy person who is away from home can be easily checked by his orher family. In this manner, various applications are conceivable.

The present invention is not limited to the above-described embodiment,and various modifications can be made as exemplary structures of theapparatuses and exemplary processes.

The foregoing structures of the information display apparatus 1, thetransmitting apparatus 100, the display apparatus 200, and the serverapparatus 70 are only examples, and additions and deletions of variousstructural elements can be made in accordance with the actuallyimplemented operations and functions.

In the information processing system shown in FIG. 9, the displayapparatus 200 obtains bio-information and the like detected by thetransmitting apparatus 100 via the server apparatus 70. Alternatively, asystem configuration including no server apparatus 70 is conceivable.That is, in a system in which the transmitting apparatus 100 and thedisplay apparatus 200 directly communicate with each other, the displayapparatus 200 can receive bio-information, kinetic information, positioninformation, and the like detected by the transmitting apparatus 100 anddisplay an avatar based on another person's bio- and kinetic informationor the avatar on a map using the position information.

The information display apparatus 1 and the display apparatus 200 eachinclude the display unit 14 (214). Alternatively, the informationdisplay apparatus 1 and the display apparatus 200 may include no displayunit and may display an avatar using another independent display device.

With regard to displaying an avatar of another person based on thisperson's bio- and kinetic information and the position of this personbased on this person's position information, it is appropriate to imposesome limits in order to protect this person's privacy.

For example, when the user of the display apparatus 200 requests todisplay an avatar and the current position of the user of thetransmitting apparatus 100, the server apparatus 70 asks the user of thetransmitting apparatus 100 to agree to display his or her avatar andcurrent position. Only when the user of the transmitting apparatus 100agrees to do so, the server apparatus 70 transmits bio-information andavatar-related information to the display apparatus 200.

Further, when detected bio-information which is the heart rate, bloodpressure, or the like shows a normal value, the bio-information may notbe used to display an avatar. Only when the detected bio-informationshows a physically abnormal value, the bio-information may be used todisplay an avatar. Accordingly, an avatar serving as a warning or alertto the user or another person may be displayed.

By constructing the transmitting apparatus 100 to be wearable by aliving body other than a human being, such as a pet, e.g., a dog or acat, the pet can serve as a subject, and its bio-information, kineticinformation, and position information can be detected. In this case, theuser of the display apparatus 200 can check the pet's state using anavatar.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

What is claimed is:
 1. A non-transitory computer readable medium havingencoded thereon executable instructions that, when executed by at leastone processor, perform a method comprising: determining a user state ofa first user based on an image of the first user captured by an imagesensor and a movement of the first user detected by an accelerationsensor and/or a gyro sensor; and controlling a real-time display of anavatar image of the first user, representing the user state of the firstuser, and an avatar image of a second user, representing a user state ofthe second user, on a display, wherein the user state of the second useris determined via an external device based on an image of the seconduser and a movement of the second user.
 2. The non-transitory computerreadable medium of claim 1, wherein the image of the second user iscaptured and/or the movement of the second user is detected via theexternal device.
 3. The non-transitory computer readable medium of claim1, wherein the avatar image of the first user represents a still state,a walking state, a running state, and/or an exercising state of thefirst user.
 4. The non-transitory computer readable medium of claim 1,wherein the movement of the first user is detected by detecting movementof a body element of the first user, including a head, an arm, a leg, ahand, and/or a finger of the first user.
 5. The non-transitory computerreadable medium of claim 1, wherein the non-transitory computer readablemedium is included within an apparatus, wherein the apparatus includes awrist watch, glasses, a headset, a hat, a helmet, a glove, and/orclothing including the apparatus.
 6. The non-transitory computerreadable medium of claim 1, wherein the method further comprisesdetermining whether a current mode of the display is a map display mode.7. The non-transitory computer readable medium of claim 6, wherein themethod further comprises, based on determining that the current mode isthe map display mode, displaying the avatar image of the first user at aposition corresponding to a current position of the first user on a mapimage.
 8. The non-transitory computer readable medium of claim 7,wherein the method further comprises, based on a change in the currentposition of the first user and/or the user state of the first user,updating a range of the displayed map image, the current position of thefirst user on the map image, and/or the avatar image of the first user.9. The non-transitory computer readable medium of claim 8, wherein themethod further comprises displaying a past avatar image of the firstuser based on a past user state of the first user.
 10. An informationprocessing method, the method comprising: determining a user state of afirst user based on an image of the first user captured by an imagesensor and a movement of the first user detected by an accelerationsensor and/or a gyro sensor; and controlling a real-time display of anavatar image of the first user, representing the user state of the firstuser, and an avatar image of a second user, representing a user state ofthe second user, on a display, wherein the user state of the second useris determined via an external device based on an image of the seconduser and a movement of the second user.
 11. The method of claim 10,wherein the image of the second user is captured and/or the movement ofthe second user is detected via the external device.
 12. The method ofclaim 10, wherein the avatar image of the first user represents a stillstate, a walking state, a running state, and/or an exercising state ofthe first user.
 13. The method of claim 10, wherein the movement of thefirst user is detected by detecting movement of a body element of thefirst user, including a head, an arm, a leg, a hand, and/or a finger ofthe first user.
 14. The method of claim 10, wherein the non-transitorycomputer readable medium is included within an apparatus, wherein theapparatus includes a wrist watch, glasses, a headset, a hat, a helmet, aglove, and/or clothing including the apparatus.
 15. The method of claim10, further comprising determining whether a current mode of the displayis a map display mode.
 16. The method of claim 15, further comprising,based on determining that the current mode is the map display mode,displaying the avatar image of the first user at a positioncorresponding to a current position of the first user on a map image.17. The method of claim 16, further comprising, based on a change in thecurrent position of the first user and/or the user state of the firstuser, updating a range of the displayed map image, the current positionof the first user on the map image, and/or the avatar image of the firstuser.
 18. The method of claim 17, further comprising displaying a pastavatar image of the first user based on a past user state of the firstuser.
 19. An information processing apparatus comprising: circuitryconfigured to: determine a user state of a first user based on an imageof the first user captured by an image sensor and a movement of thefirst user detected by an acceleration sensor and/or a gyro sensor; andcontrol a real-time display of an avatar image of the first user,representing the user state of the first user, and an avatar image of asecond user, representing a user state of the second user, on a display,wherein the user state of the second user is determined via an externaldevice based on an image of the second user and a movement of the seconduser.
 20. The apparatus of claim 19, wherein the image of the seconduser is captured and/or the movement of the second user is detected viathe external device.